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EXAMINING  THE  EFFECT  OF  INFORMATION  ORDER  ON  EXPERT  JUDGMENT 


EXECUTIVE  SUMMARY 


Requirement : 

SubstantialAgesearch  indicates  chat  humans  use  heuristics  to  make 
inferences  and  that,  depending  on  task  characteristics,  these  heuristics 
can  lead  to  inconsistencies  and  errors  in  Judgment — that  is,  cognitive 
biases.  Most  of  this  research  has  been  performed  with  university  students 
performing  tasks  requiring  logical  thinking  but  not  expertise  in  a  par¬ 
ticular  substantive  area.  Our  concern  is  in  determining  whether  heuristics 
can  lead  to  cognitive  biases  among  experienced  personnel  performing  their 
substantive  task.  In  particular,  we  examined  whether  information  order  and 
response  mode  could  affect  the  judgments  of  Army  air  defense  operators. 


Procedure : 


A  wi thin-subject  factorial  experiment  was  performed  in  December, 

1989,  with  63^ Armyjair  defense  operators^stationed  at  Fort  Bliss,  Texas. 
Five  factors  were  varied  using  a  paper-and-pencil  format:  (1)  whether  the 
information  about  an  incoming  aircraft  of  unknown  identity  was  presented 
sequentially  or  all  at  once,  (2)  the  order  with  which  the  same  information 
was  presented,  (3)  whether  the  first  piece  of  information  supported  the 
conclusion  that  the  aircraft  was  friendly  or  hostile,  (4)  whether  the  first 
piece  of  information  was  strong  or  weak,  and  (5)  whether  subsequent 
Information  negating  the  first  piece  of  Information  was  strong  or  weak. 

The  dependent  variable  was  a  probability  estimate  for  the  identity  of  the 
aircraft  (friend  or  hostile)  after  receiving  all  information. 


Findings : 

— ^Information  order  and  response  mode  Interacted  to  affect  tue  Army  air 
defense  operators'  judgments.  When  information  was  presented  sequentially 
and  a  probability  estimate  was  obtained  after  each  piece  of  information, 
participants  gave  different  probability  estimates  of  whether  an  unknown 
aircraft  was  friendly  or  hostile,  depending  on  the  order  with  which  the 
same  information  was  presented. ->^In  contrast,  there  was  no  order  effect 
when  the  information  was  presented  all  at  onceT^  These  results  support  the 
predictions  of  the  Hogarth-Einhom  belief  updating  model.  There  were, 
however,  large  individual  differences. 


Utilization  of  Findings: 

Discussion  with  senior- level  Army  air  defense  personnel  indicates 
that  the  results  may  have  implications  for  (1)  training  air  defense 
operators,  and  (2)  designing  information  displays  for  future  air  defense 
systems.  The  case  would  be  stronger  if  the  results  were  obtained  using  (i) 

vfci 


Preceding  Page  Blank 


actual  ainulatora,  and  (ii)  if  wa  had  Xnvestigatad  whether  ehangee  in 
Identification  Judgment a  also  affactad  engagement  behavior.  Preparation  of 
sueh  a  atudy  ia  currently  underway. 
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EXAMINING  THE  EFFECT  OF  INFORMATION  ORDER  ON  EXPERT  JUDGMENT 


INTRODUCTION 


Substantial  research  reviewed  in  Hogarth  (1987),  Kahneman  at  al. 
(1982),  Nisbett  nnd 'Ross  (1980),  etc.  indicates  that  humans  use  heuristics 
to  make  inferences  and  that,  depending  on  task  characteristics,  these 
heuristics  can  lead  to  inconsistencies  and  errors  in  judgment;  that  is, 
cognitive  biases.  Although  there  are  exceptions  (e.g.,  see  Flschhoff  at 
al.,  1978;  Tversky  and  Kahneman,  1971),  most  of  this  research  has  been 
performed  with  university  students  performing  tasks  requiring  logical 
thinking  but  not  expertiee  in  a  particular  substantive  aree.  Our  concern 
is  in  determining  whether  heuristics  can  lead  to  cognitive  biases  among 
experienced  personnel  performing  their  substantive  task.  In  particular, 
the  experiment  reported  herein  examined  whether  information  order  and 
response  mode  could  affect  the  judgments  of  Army  air  defense  operators. 

Einhom  and  Hogarth  (1987)  and  Hogarth  and  Einhom  (1989)  developad  a 
belief  updating  model  attempting  to  explain  the  diverse  findings  in  the 
literature  regarding  the  effect  of  information  order.  This  model  assumes 
an  anchoring  and  adjustment  process  that  depends  on  four  task  characteris¬ 
tics:  the  amount  of  information  presented,  whether  the  information  is 
simple  or  complex,  the  order  in  which  the  information  is  presented,  and 
whether  a  probability  estimate  is  obtained  after  presenting  each  piece  or 
ell  the  information.  Since  the  experiment  reported  herein  Involved  only  e 
short  series  of  information  and  was  simple  for  trained  personnel,  we 
consider  only  their  model's  prediction  for  information  order  end  response 
mode . 


Specifically,  when  information  is  presented  sequentially  and  a 
probability  estimate  is  obtained  after  each  piece  of  information,  the 
Hogarth- Einhom  model  predicts  that  people  will  anchor  on  the  current 
position  end  adjust  it  on  the  basis  of  the  strength  and  direction  (positive 
or  negative)  of  each  new  piece  of  information.  Since  each  new  piece  of 
information  creates  a  new  anchor,  recent  information  is  weighted  more  than 
prior  information,  thereby  resulting  in  a  recency  effect.  Moreover,  they 
hypothesized  that  the  larger  the  anchor,  the  greeter  the  impact  of  the  same 
piece  of  negative  evidence.  Conversely,  the  smaller  the  anchor,  the 
greeter  the  impact  of  the  same  piece  of  positive  evidence.  Consequently, 
the  order  in  which  the  same  positive  and  negative  evidence  is  presented  is 
predicted  to  result  in  different  final  probability  estimates. 

In  contrast,  when  information  is  presented  all  at  ones  and  a  proba¬ 
bility  estimate  is  obtained  at  that  time,  the  Hogarth -Einhom  model 
predicts  that  people  will  anchor  on  the  piece  of  information  presented 
first  end  adjust  it  on  the  basis  of  the  aggregate  Impact  of  all  subsequant 
Information  in  support  of  the  initial  information  (l-e.,  positive  evidence) 
end  against  it  (l.e.,  negative  evidence).  Since  the  model  predicts  that 
subsequent  information  will  be  considered  in  the  aggregate,  the  order  of 
positive  and  negative  evidence  is  predicted  to  have  no  effect  on  the  final 
probability  estimates. 
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Thus  far,  thara  Is  minimal  empirical  support  for  this  thaoratleal 
nodal  using  trained  personnel  performing  their  substantive  task.  Although 
Einhom  and  Hogarth  (1987)  and  Hogarth  and  Einhorn  (1989)  present  empirical 
results  supporting  the  model,  all  the  experiments  mere  with  university 
students  performing  general  tasks — not  trained  personnel  performing  their 
substantive  task.  This  is  also  true  of  most  of  the  studies  they  reviewed, 
as  well  as  others  (s.g.,  Hamm,  1987)  supporting  some  of  their  predictions.  - 
Asare  (1990),  Ashton  and  Ashton  (1988;  1990),  end  Meisser  (1990)  do, 
however,  present  empirical  results  consistent  with  the  model's  predictions 
using  professional  auditors.  Serfaty  et  al.  (IV OS)  also  present  results 
obtained  from  Army  officers  showing  an  order  effect  consistent  with  the 
model's  predictions. 

There  is  also  research  with  trained  personnel  predicting  results  dif- 
ferent  from  the  Hogarth -Einhom  model.  In  a  series  of  experiments  where 
Information  was  presented  sequentially,  Tolcott  et  al.  (1988,  1989a,  1989b) 
showed  that  Army  intelligence  analysts  exhibited  a  "confirmation  bias,” 
whereby  they  gave  greater  weight  to  positive  evidence  confirming  their 
initial  hypothesis  about  the  enemy's  course  of  action  than  to  negative 
evidence  disconfirmlng  their  early  judgment.  An  anchoring  and  adjustment 
model  was  inferred  by  the  researchers  to  explain  the  findings,  but  one 
where  (1)  the  initial  information  was  overweighted,  and  (2)  subsequent 
information  was  given  less  weight  the  later  it  was  received.  Consequently, 
that  research  would  predict  a  primacy  effect  when  information  is  presented 
sequentially,  not  the  recency  effect  predicted  by  the  Hogarth- Einhom 
model . 


It  should  be  noted,  however,  that  the  procedures  used  in  the  studies 
by  Tolcott  et  al.  differed  from  those  used  by  Hogarth  and  Einhom  in  three 
important  ways.  First,  the  initial  information  was  embedded  in  complex, 
ambiguous  scenarios  in  the  Tolcott  et  al.  studies;  straightforward,  one- 
paragraph  scenarios  ranging  between  68  to  109  words  were  used  in  the 
Hogarth- Einhom  scenarios.  Second,  in  the  Tolcott  et  al.  studies,  par¬ 
ticipants  received  groups  of  information  sequentially.  For  example, 
participants  in  Tolcott  et  al.  (1989b)  received  three  intelligence  reports 
containing  fifteen  pieces  of  information  in  each  report.  Participants  in 
the  Hogarth -Einhorn  studies  received  only  one  piece  of  information  each 
time  they  were  provided  subsequent  information.  Third,  the  subsequent 
information  presented  at  each  time  period  in  the  Tolcott  et  al.  studlas  was 
ambiguous.  For  example,  in  Tolcott  et  al.  (1989b),  three  of  the  fifteen 
pieces  of  Information  supported  one  hypothesis,  three  the  other  hypothesis, 
and  nine  pieces  of  information  were  neutral.  In  the  Hogarth- Einhom 
studies,  subsequent  information  varied  in  strength  but  it  always  supported 
one  hypothesis  or  the  other. 

Before  turning  to  the  Hethod  section,  we  want  to  point  out  that  order 
effect#  may  ba  an  important  issue  in  the  Army  air  defense  task.  Army  air 
defense  operators  must  identify  incoming  aircraft  as  frisndly  or  hostile, 
and  then  engage  hostiles  under  conditions  of  savers  strsss  and  tins 
pressure.  The  identification  is  based  on  information  of  various  degrees  of 
diagnosticlty.  Information  can  be  received  sequentially  end  in  various 
orders.  The  Interfaces  for  newer  air  defense  systems  can  display  much  of 
the  essential  Initial  information  at  once  and  continue  to  display  this 
Information  as  new  information  is  recsived,  but  older  systems  can  not  and 
represent  more  of  a  sequential  information  flow.  Moreovar,  some  informs- 
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tion  (e.g. ,  from  headquarters)  is  not  entered  Into  the  automated  system, 
just  passed  to  the  operator  via  communications .  Zn  sum,  the  Army  air 
defense  task  was  considered  both  appropriate  and  important  for  studying  the 
effect  of  Information  order  on  expert  judgment. 
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METHOD 


Participants 

Sixty* three  Army  air  defense  operators  from  four  battalions  (three 
PATRIOT,  one  HAWK)  stationed  at  Fort  Bliss,  Texas,  participated  in  the 
experiment,  which  was  conducted  in  mid-December  1989.  All  participants  had 
completed  their  training  and  participated  in  Army  air  defense  exercises 
simulating  combat  conditions  in  field  settings,  as  well  as  on  training 
simulators,  prior  to  participating  in  the  study.  Examination  of  the 
background-  inf ormation  sheets  completed  by  the  participants  at  the  time  of 
the  study  indicated  that  thirty  of  the  participants  were  enlisted  personnel 
and  thirty- three  were  officers.  The  mean  air  defense  experience  of  the 
enlisted  personnel  was  4.1  years,  with  a  range  from  0.5  to  14  years.  The 
mean  air  defense  experience  of  the  officers  was  2.2  years,  with  a  range 
from  0.5  to  7.5  years. 


CfifilgQ 

There  were  five  independent  variables.  The  first  independent 
variable  was  the  order  in  which  positive  and  negative  information  was 
presented  to  the  participants.  Positive  information  was  defined  as 
information  consistent  with  the  conclusion  one  would  reach  on  the  basis  of 
the  first  piece  of  information;  negative  information  was  inconsistent  with 
the  conclusion.  In  Order  #1,  the  second  piece  of  information  was  positive 
(P)  and  the  third  piece  of  information  was  negative  (N) .  In  Order  #2,  the 
second  piece  of  information  was  negative  and  the  third  piece  of  information 
was  positive.  Only  three  pieces  of  information  were  presented  about  each 
case  (i.e.,  unknown  aircraft).  The  positive  and  negative  pieces  of 
information  were  always  the  second  and  third  pieces  of  information.  In  all 
cases,  two  pieces  of  information  would  point  toward  one  conclusion  (e.g. , 
the  aircraft  was  friendly)  and  the  other  piece  of  information  would  point 
toward  the  other  conclusion  (e.g.,  hostile). 

The  second  independent  variable  was  response  mode:  information  was 
presented  either  sequentially  with  a  probability  estimate  being  obtained 
after  each  piece  of  information,  or  all  at  once  with  the  probability 
estimate  being  obtained  at  that  time.  The  third  independent  variable  was 
whether  the  first  piece  of  information  supported  the  conclusion  that  the 
unknown  aircraft  was  friendly  or  hostile.  The  fourth  independent  variable 
was  whether  the  first  piece  of  information  was  strong  or  weak.  Based  on 
pilot  testing,  a  strong  piece  of  Information  was  defined  as  that  with  a 
probability  2;  0.8  in  support  of  friend  or  hostile.  A  we«k  piece  of 
information  had  a  probability  >0.5  but  £  O.C.  The  fifth  Independent 
variable  was  whether  the  negative  information  was  strong  or  weak. 

In  sum,  our  experiment  was  the  following  crossed,  factorial  design: 

2  (order)  x  2  {response  mode)  x  2  (conclusion)  x  2  (strength  of  first  piece 
of  information)  x  2  (strength  of  negative  information).  The  last  three 
independent  variables  defined  eight  hypothetical  aircraft.  We  defined 
sixteen  critical  cases  by  varying  the  erder  in  which  the  last  two  places  of 
information  ware  presented  about  these  aircraft.  The  experiment  was  a 
completely  within- subject  design  because  all  participants  evaluated  all 
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cases.  The  dependent  variable  for  all  cases  was  a  probability  as Cleats 
that  the  unknown  aircraft  was  friendly  or  hostile. 

Participants  were  also  asked  to  write  the  reason  for  their  final 
probability  estimate  when  the  information  was  presented  sequentially.  This 
was  done  in  an  attempt  to  understand  their  decision  process.  Pilot  testing 
Indicated  that  it  would  be  too  onerous  to  obtain  written  rationale  every 
time  the  participants  made  their  probability  estimates,  tfe  would  have 
preferred  to  use  talk-aloud  protocols,  but  they  were  not  feasible  because 
the  participants  had  to  be  run  in  groups.  As  it  turned  out,  the  par¬ 
ticipants  seldom  provided  written  rational  for  their  judgments  and  when  it 
was  provided,  it  was  often  too  cryptic  to  be  useful.  Consequently,  no 
analysis  was  performed  on  the  written  rationale. 


Information  about  all  the  cases  defined  by  our  design  was  presented 
in  a  booklet  format.  One  book  contained  all  the  cases  when  the  information 
was  presented  all  at  once.  Two  books  were  used  to  present  all  the  cases 
when  the  information  was  presented  sequentially.  Two  books  were  used  to 
lessen  the  visual  impact  of  the  participants'  task.  The  order  in  which 
participants  completed  the  books  was  randomly  determined. 

The  Mall-at-onceN  book  and  the  '"sequential"  books  contained  36  cases. 
The  same  cases  were  presented,  and  in  the  same  order,  in  each  book.  The  36 
cases  were  composed  of  the  16  cases  comprising  the  above  design  (not 
considering  response  mode)  and  20  other  cases.  The  latter  were  included  to 
minimize  the  probability  that  participants  would  remember  their  answers  to 
the  16  design  cases.  The  16  cases  were  randomly  distributed  throughout  the 
books . 


Participants  completed  the  books  individually.  The  books  were 
completed  in  four  group  sittings — one  for  each  of  the  four  air  defense 
battalions  participating  in  the  study.  The  groups  varied  in  size,  ranging 
from  about  ten  to  twenty  participants.  Participants  completed  the  backV 
ground  Information  form  prior  to  completing  the  books. 


Hymhm* 

Consistent  with  the  Hogarth- Einhom  theoretical  model,  we  predicted 
(1)  a  recency  effect  when  the  information  was  presented  sequentially  and  a 
probability  estimate  was  received  after  each  piece  of  information,  and  (2) 
a  no -order  effect  when  the  Information  was  presented  all  at  once  and  a 
probability  estimate  was  obtained  at  that  time.  For  the  recency  effect, 
negative  information  should  be  weighted  more  when  the  anchor  is  large; 
positive  information  should  be  weighted  more  when  the  anchor  is  small. 

This  was  hypothesized  to  result  in  a  final  probability  estimate  that  was 
higher  when  negative  information  was  followed  by  positive  information 
(i.e.,  the  NP  order)  than  when  positive  information  was  followed  by 
negative  information  (i.e.,  the  PN  order).  This  is  represented  pictorially 
in  Figure  1  by  what  Einhom  and  Hogarth  (1987)  called  a  "fishtail.” 
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Figure  1.  Notional  representation:  Predictions  based 
on  Hogarth- Einhorn  model. 


When  all  the  information  is  presented  at  once,  the  Hogarth-Einhom 
model  predicts  that  people  will  anchor  on  the  first  piece  of  information  in 
the  series  and  adjust  it  on  the  basis  of  the  aggregate  impact  of  the 
remaining  information.  For  our  task  in  which  participants  receive  only 
three  pieces  of  information,  there  should  be  no  differences  in  the  final 
probability  estimates  due  to  order  of  the  last  two  pieces  of  information. 

A  notional  representation  of  this  prediction  is  also  presented  in  Figure  1. 

There  were  two  alternative  hypotheses.  The  first  one,  based  on  the 
research  by  Tolcott  et  al.  demonstrating  a  "confirmation  bias,"  was  that 
there  would  be  a  primacy — not  recency-effect  when  information  was  pre¬ 
sented  sequentially.  This  hypothesis  was  also  based  theoretically  on  an 
anchoring  and  adjustment  model,  but  one  where  the  initial  information 
(i.c.,  anchor)  was  overweighted  and  subsequent  information,  particularly  if 
it  was  negative,  was  given  less  weight  the  later  it  was  received.  This  was 
predicted  to  result  in  a  higher  final  probability  estimate  when  positive 
information  was  followed  by  negative  information  (the  FN  order)  than  when 
negative  information  was  followed  by  positive  information  (the  HP  order). 
The  notional  representation  of  this  hypothesis  is  presented  in  Figure  2. 

The  "all  at  once"  condition  did  not  exist  in  the  studies  by  Tolcott  et  al.; 
consequently,  no  predictions  were  made. 
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Figure  2.  Notional  representation:  Predictions  based  on 
Tolcott  et  ai.'s  research. 


The  second  alternative  hypothesis  was  that  there  would  be  no  order 
effects  regardless  of  how  Information  was  presented.  This  was  not  only  the 
proverbial  null  hypothesis,  it  was  the  first  author's  actual  hypothesis. 
This  hypothesis  was  based  on  the  fact  that  most  of  the  previous  research 
was  performed  with  university  students.  The  first  author  did  not  think 
that  highly  trained  personnel  would  give  different  final  probability 
estimates  to  the  same  Information  when  It  was  presented  sequentially  simply 
because  one  had  switched  the  order  of  the  last  two  (of  only  three)  pieces 
of  information. 


A  within ‘subject  2  (order)  x  2  (response  mode)  x  2  (conclusion)  x  2 
(strength  of  first  piece  of  information)  x  2  (strength  of  negative  informa¬ 
tion)  Analysis  of  Variance  (ANOVA)  was  performed  using  (1)  the  final 
probability  estimates  after  all  three  pieces  of  information  were  presented 
sequentially,  and  (2)  the  sole  probability  estimates  when  the  information 
was  presented  all  at  once.  The  probability  estimates  for  each  case  were 
coded  in  the  direction  specified  by  the  "conclusion"  independent  variable. 
For  example,  a  0.60  probability  estimate  for  either  a  friendly  or  a  hostile 
aircraft  was  coded  as  0.80.  Probabilities  were  multiplied  by  100  when  the 
data  were  coded  to  avoid  potential  errors  due  to  forgetting  decimal  points. 
(Note :  This  resulted  In  large  "sums  of  squares*  and  •mean  square*  terms  in 
ths  ANOVA.) 
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RESULTS 


The  ANOVA  shoved  vain  effects  for  each  of  the  five  Independent 
variables.  First,  there  was  a  mein  effect  for  the  first  piece  of  informa¬ 
tion:  £(1,62)  -48.4,  MS,  -  203.78,  £<  .001.  The  final  probabilities 
were,  on  the  average,  higher  when  the  first  piece  of  information  was  strong 
{*rrts  “  «S9)  than  weak  (xMlW  -  .64).  Second,  there  was  a  main  effect  for 
strength  of  negative  information:  £(1,62)  -  129.69,  US*  -  S37.19,  £  < 

.001.  The  final  probabilitiesjwere  higher  when  negative  information  was 
weak  (St,,*  -  .72)  than  strong  <Xg.8  -  .60):  £(1.62)  -  129.69,  MS#  -  537.19, 

£  <  .001.  These  main  effects  Indicate  that  the  manipulation  of  these 
Independent  variables  was  successful.  Third,  there  was  a  main  effect  for 
conclusion:  £(1,62)  -  19.58,  MS,  -  2481.47,  £  <  .001.  Friendly  aircraft 
received  e  higher  final  probability  (xr  -  .71)  than  hostile  aircraft  (x*  - 
.60).  This  was  not  a  manipulation  check  because  all  cases  were  coded  in 
the  direction  of  the  "conclusion"  independent  variable.  Instead,  the 
results  indicate  that  the  participants  had  more  confidence  in  the  identity 
of  the  friendly  than  hostile  aircraft  on  the  basis  of  the  information 
provided  to  them. 

We  also  obtained  main  effects  for  response  mode  [£(1,62)  -  5.85,  MS, 

-  857.14,  £  <  .02]  and  order  [£(1,62)  -  29.84,  US*  -  1198.91,  £  <  .001]. 

The  final  probabilities  were  higher_when  information  was  presented  all  at 
once  (Xq  —  .68)  than  sequentially  (xs  -  .65).  And  the  final  probabilities 
were  higher  when  positive  information  followed  negative  information  (xgp  - 
.71)  than  when  it  preceded  negative  Information  (xm  -  .60). 

There  were  a  number  of  significant  interactions.  Of  particular 
Interest,  given  our  hypotheses,  was  that  we  obtained  a  significant  order  by 
response  mode  interaction:  £(1,62)  -  20.47,  US,  -  1124.6;  £  <  .001.  As 
shown  in  Figure  3,  the  mean  final  probability  estimates  were  essentially 
identical  when  information  was  presented  all  at  once.  In  contrast,  the 
mean  final  probability  estimates  were  significantly  higher  for  the  NP  order 
(x  -  0.73)  than  the  FN  order  (x  -  0.58)  when  the  information  vaa  presented 
sequentially . 


8 


INFORMATION  ITEM 


Figure  3.  Mean  probability  estimates  showing  support 
for  the  interaction  effect  predicted  by  the 
Hogarth -Einhorn  model. 


These  results  support  the  predictions  of  the  Hogarth-Einhorn  model. 

No  order  effect  was  obtained  when  the  information  was  presented  all  at 
once;  an  order  effect  was  obtained  when  the  information  was  presented 
sequentially.  Further  examination  of  Figure  3  shows  that  when  the  informa¬ 
tion  was  presented  sequentially,  positive  information  had  a  larger  impact 
the  smaller  the  anchor  (x  -  +0.29  v».  x  -  +0.11),  consistent  with  the 
model's  predictions.  However,  in  contrast  with  the  model's  predictions, 
negative  information  did  not  have  a  larger  impact  the  larger  the  anchor 
(xpn  -  >0.24  vs.  xMp  -  *0.26).  Although  there  were  minor  differences,  this 
finding  was  obtained,  as  shown  in  Table  1,  for  most  of  the  eight  com¬ 
parisons  defined  by  the  2  (conclusion)  x  2  (strength  of  first  piece  of 
information)  x  2  (strength  of  negative  information)  portion  of  the  design. 
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Table  1 


The  Difference  in  the  Mean  Probability  Estimate 
Prior  to  and  After  Receipt  of  Negative  Information 
in  the  Sequential  Information  Presentation  Condition 


SIMW.W1 

KgJIEg 

ggurm 

CONCLUSION:  FRIEND 

INITIAL  SUPPORT:  STRONG 

-.22 

-.25 

-.21 

-.21 

INITIAL  SUPPORT:  WEAN 

-.20 

-.21 

-.18 

*.20 

CONCLUSION*.  HOSTILE 

INITIAL  SUPPORT:  STRONG 

-.34 

-.U 

-.15 

-.21 

INITIAL  SUPPORT:  WEAK 

-.44 

-.38 

-.19 

-.21 

There  were  six  other  interactions  that  reached  traditional  sig¬ 
nificance  levels.  Five  involved  either  the  response  mode  or  order  indepen¬ 
dent  variables.  Ve  consider  them,  in  turn. 

First,  there  was  a  conclusion  x  strength  of  negative  information 
interaction:  £(1,62)  —  53.24,  MS,  -  538.54,  p  <  .001.  This  occurred 
because  strong  negative  Information  bad  a  much  greater  effect  on  the  final 
probabilities  when  the_aircraft  was_  hostile  (xB>M:8  -  .52,  Xh,j(iM  -  .71)  than 
when  it  was  friendly  (xFiB.s  -  .69,  xF>B:W  —  .74).* 

Second,  there  was  a  response  mode  x  strength  of  negative  information 
interaction:  £<1,62)  -  3.92,  US,  -  173.28,  a  -  .05.  This  occurred  because 
the  effect  of  strong  versus  weak  negative  information  was  slightly  more 
pronounced  when  the  information  was  presented  all  at  once  (Xo,N:s  "  .61, 
Xo.k:h  -  *74,  diff.  -  .13)  than  sequentially  (Xa,*^  -  .59,  -  -70,  diff. 


Third,  there  was  an  order  x  strength  of  negative  information  interac¬ 
tion:  £(1,62)  *  6.52,  MS,  -  100.21,  a  <  *02.  This  occurred  because  the 
obtained  order  effect  was  slightly  more  pronounced  when  the  negative 
information  was^  strong  (xWilf}8  -  .55,  -  .65,  diff.  -  .10)  than  weak 

(Kw.hsw  ■  *69,  Xsp.stw  "  *76.  diff.  .07). 

Fourth,  there  was  a  conclusion  x  order  x  strength  of  negative 
Information  interaction:  £(1,62)  -  8.13,  M£,  -  80.113,  a  <  .01.  This 
occurred  because  the  order  x  strength  of  negative  information  Interaction 
was  somewhat  more  pronounced  for  hostile  than  friendly  aircraft  when  the 
negative  information  was  strong,  but  not  when  it  was  weak. 

Fifth,  there  was  a  response  mode  x  conclusion  x  order  x  negative 
information  interaction:  £(l,62)  -  8.32,  tl£#  *  72.94,  a  <  *01.  This 
occurred  because  the  conclusion  x  order  x  strength  of  negative  information 
interaction  occurred  primarily  when  information  was  presented  sequentially, 
not  all  at  once. 

Lastly,  there  was  a  strength  of  first  piece  of  information  x  con¬ 
clusion  x  order  x  strength  of  negative  information  interaction:  £(1,82)  - 
6.96,  US,  -  76.49,  a  •  *01.  This  occurred  because  the  conclusion  x  order  x 
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strength  of  negative  information  interaction  occurred  when  the  first  piece 
of  information  was  weak,  but  not  when  it  was  strong. 

t 

Ve  examined  the  amount  of  variation  (R2)  in  the  final  probability 
estimates  accounted  for  by  the  statistically  significant  effects.  Tour 
points  are  important  to  note  here.  First,  we  found  that  the  significant 
effects  due  to  order  and  response  mode  accounted  for,  in  total,  a  small 
proportion  of  the  total  variation  in  the  participants'  final  probability 
estimates;  R2  -  0.074.  Tor  example,  the  R2  for  the  order  by  response  mode 
Interaction  was  0.025;  the  R2  for  the  order  main  effect  vas  0.039.  Second, 
the  "strength-of*negative-informatiori"  main  effect  (R2  -  0.076)  and  the 
"conclusion"  (i.e.,  friend  versus  hostile)  main  effect  (R2  -  0.053) 
accounted  for  more  variance  than  any  slngla  "order"  or  "response-mode" 
affect.  This  suggests  that  these  factors  had  s  stronger  effect  on  Army  elr 
defense  operators'  probability  estimates  than  did  order  or  response  mode. 
Third,  the  R2  for  each  of  the  third-  and  fourth- order  interactions  was 
.001.  Although  they  achieved  traditional  lavale  for  statistical  sig¬ 
nificance,  these  effects  were  small.  And,  fourth,  all  significant  (i.e., 

&  0.05)  effects  accounted  for,  in  total,  24.7  percent  of  the  total  varia¬ 

tion.  This  indicates  that  most  of  the  variation  in  the  participants'  final 
probability  estimates  was  due  to  subjects  snd  not  the  manipulated  indepen¬ 
dent  variables. 

Three  post-hoc  analyses  were  performed  in  an  attempt  to  better  under¬ 
stand  the  effect  of  individual  differences  in  the  participants'  probability 
estimates.  First,  an  ANOVA  was  performed  using  participants'  rank  as  a 
sixth  Independent  variable.  As  the  reader  will  remember,  thirty  of  the 
participants  were  enlisted  personnel  end  thirty- three  were  officers. 

The  ANOVA  resulted  in  three  interesting  findings.  First,  there  was  s 
main  effect  for  rank:  £(1,61)  -  7.05,  M*  -  1445.596,  and  p  -  0.01. 
Officers  had  a  larger  final  mean  probability  estimate  (0.69)  than  enlisted 
personnel  (0.645).  Second,  there  was  a  significant  rank-by-order  interac¬ 
tion:  £(1,61)  ~  5.11,  US,  -  1124.325,  end  g  -  0.026.  The  difference  in 
the  mean  probability  estimate  for  the  two  order  conditions  vas  smaller  for 
officers  (0.06)  than  enlisted  personnel  (0.12).  And,  third,  the  rank-by- 
order-by-response-mode  interaction  approached  significance:  £(1,61)  - 
3.67,  “  1078.202,  £  -  0.057.  As  Figure  4  illustrates,  the  difference 

in  the  final  mean  probabilities  estimates  for  the  NF  end  FN  orders  was 
larger  for  enlisted  personnel  (0.22)  than  officers  (0.10)  when  information 
was  presented  sequentially.  In  contrast,  the  difference  was  small  (0.03) 
and  identical  for  enlisted  and  officer  personnel  when  information  was 
presented  all  at  once.  However,  all  significant  effects  Incorporating  rank 
accounted  for  little  additional  variation  in  the  participants'  probability 
estimates  (R2  <  0,03). 
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Figure  4.  Mean  probability  estimates  showing  larger  order  effects 
for  enlisted  personnel  than  officers  when  information 
was  presented  sequentially. 


The  second  post-hoc  analysis  was  the  addition  of  a  seventh  fac¬ 
tor — experience  level — to  the  ANOVA.  This  factor  had  two  conditions:  (1) 
less  than  or  equal  to  one  year's  air  defense  experience,  and  (2)  more  than 
one  year's  experience.  There  were  17  officers  and  9  enlisted  personnel 
with  less  than  one  year's  experience,  and  14  officers  and  21  enlisted 
personnel  with  more  than  a  year's  experience;  two  of  the  officers  did  not 
indicate  their  years  of  experience.  Neither  the  experience  main  effect, 
experience -by- rank  interaction,  nor  any  of  the  experlence-by-order  interac¬ 
tions  Involving  less  than  five  factors  were  significant  at  p  <  0.05  level. 

The  third  post -hoc  analysis  examined  the  extent  to  which  Individual 
participants  exhibited  order  effects  when  information  was  presented 
sequentially  and  all  at  once,  respectively.  This  was  accomplished  by 
computing  the  difference  in  the  final  probability  estimates  for  the  NP  and 
PN  orders  when  information  was  presented  sequentially,  and  the  sole 
probability  estimates  when  information  was  presented  all  at  once,  for  the 
eight  cases  defined  by  the  2  (conclusion)  x  2  (strength  of  first  piece  of 
Information)  x  2  (strength  of  negative  information)  portion  of  the  design. 
For  the  purposes  of  this  analysis,  a  positive  difference  was  classified  as 
supporting  an  order  effect. 

Table  2  shows  the  differences  when  the  information  was  presented 
sequentially.  As  can  be  seen,  thirty-seven  percent  <N  —  11)  of  the 
enlisted  personnel  and  nine  percent  (N  -  3)  of  the  officers  showed  an  order 
effect  for  all  eight  comparisons.  The  mean  effect  for  the  eleven  enlisted 
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personnel  was  0.45  with  a  naan  standard  deviation  of  0.118;  tha  naan  off act 
for  tha  thraa  officers  was  0.72  with  a  naan  standard  deviation  of  0.084. 
These  data  naan  that,  on  tha  average,  these  fourteen  part&olpants  switched 
their  hypothesis  as  to  whether  the  unknown  aircraft  was  friendly  or 
hostile,  based  on  the  order  of  the  sane  lnforaation. 


Table  2 

Individual  Difference  Date  Supporting  an  Order  Effect 

when  Information  was  Presented  Sequentially  and  a 

"Probability  "Estimate  was  Obtained  after  each  Piece  of  Information 


#  OF  COMPARISONS 
FOR  WHICH  SUBJECT 
SHOWED  EFFECT 

ENUSTED 

OFFICERS  f 

n 

D 

AVERAGE 
EFFECT  OVERALL* 

D 

fl 

AVERAGE 
EFFECT  OVERALL* 

AVERAGE 

STD 

8 

ii 

37 

.45 

.118 

3 

9 

.72 

.084 

7 

i 

3 

.34 

.239 

5 

15 

.13 

.109 

6 

3 

10 

.29 

.225 

1 

3 

.15 

.138 

5 

4 

13 

.12 

.143 

1 

3 

.03 

.147 

4 

2 

7 

.03 

.098 

8 

24 

.05 

.114 

3 

3 

10 

.01 

.128 

6 

18 

.02 

.094 

2 

5 

17 

-.03 

.103 

5 

15 

-.04 

.112 

1 

n 

El 

0 

... 

4 

12 

-.01 

.069 

0 

- 

3 

-.06 

.067 

R 

H 

0 

— 

N - 30  100%  7-. 22  7- .132  N-33  99%  7-  .10  7- .103 


•POSITIVE  NUMBERS;  DIRECTION  SUPPORTING  ORDER  EFFECT 


Further  examination  of  Table  2  indicates  the  other  extreme  as  well. 
Specifically,  almost  one* third  of  the  enlisted  personnel  and  almost  one- 
half  of  the  officers  failed  to  show  an  order  effect  for  at  least  half 
(i.e.,  four  or  more)  of  the  comparisons.  Moreover,  the  mean  difference  in 
their  NP  and  PN  orders  was  essentially  zero  indicating  that,  even  when  they 
occurred,  the  order  effects  were  very  weak  for  these  participants. 

Table  3  shows  the  differences  in  the  NP  and  PN  orders  when  the 
information  was  presented  all  at  once.  As  can  be  seen,  some  participants 
showed  an  order  effect  more  consistently  than  others.  However,  most  of  the 
differences  are  close  to  zero,  indicating  that  the  effect,  even  when  it 
occurred,  was  <juite  weak  when  the  information  was  presented  all  at  once. 
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Tabid  3 


Individual  Difference  Data  Supporting  an  Order  Effect 
vhen  Info mat ion  was  Presented  All  at  Once  and  a 
Probability  Estimate  was  Obtained  at  that  lime 
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•POSITIVE  NUMBERS:  DIRECTION  SUPPORTING  ORDER  EFFECT 
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DISCUSSION 


The  results  of  the  experiment  reported  herein  with  trained  personnel 
performing  their  eubstentive  tack  support  the  predictions  of  the  Hogarth- 
Einhom  modal  for  belief  updating.  Consistent  with  their  anchoring  and 
adjustment  model,  (1)  an  order  effect  was  obtained  when  Information  mas 
presented  sequentially  and  a  probability  estimate  was  obtained  after  each 
piece  of  information,  and  (2)  a  no  order  effect  was  obtained  when  informa¬ 
tion  was  presented  all  at  once  and  the  probability  estimate  was  obtainad  at 
that  time. 

The  obtained  findings  are  consistent  with  those  of  (1)  Serfaty  at  al. 
(1989),  who  studied  military  situation  assessment  judgments,  and  (2)  Asare 
(1990),  Ashton  and  Ashton  (1988;  1990),  and  Messier  (1990),  all  of  whom 
studied  the  judgments  of  auditors.  For  clarification  purposes,  we  want  to 
note  that  the  latter  studies  used  the  term  "presentation  mode"  to  refer  to 
what  Hogarth  and  Einhom  (1989)  call  "response  mode.”  Actually,  presenta¬ 
tion  mode  and  response  mode  ere  confounded  in  these  studies  and  ours. 

Future  research  should  attempt  to  disentangle  these  variables  by,  for 
example,  presenting  information  sequentially  without  eliciting  a  probabil¬ 
ity  estimate  for  each  piece  of  information.  We  have  chosen  to  use  the  term 
"response  mode"  instead  of  "presentation  mode"  for  consistency  with  Hogarth 
and  Einhorn's  (1989)  presentation. 

The  obtained  order  effect  reported  herein  and  elsewhere  suggests 
that,  when  Information  is  presented  sequentially  and  a  probability  estimate 
is  obtained  each  time,  people  anchor  on  the  current  (not  Initial)  position 
and  adjust  it  on  the  basis  of  the  strength  and  direction  (positive  or 
negative)  of  each  new  piece  of  information.  This  results  in  a  recency 
effect.  These  findings  differ  from  the  predictions  based  on  the  research 
by  Tolcott  et  el.  (1988,  1989a,  1989b)  demonstrating  a  confirmation 
bias — i.e.,  primacy  effect.  However,  as  we  noted  in  the  Introduction,  task 
differences  may  well  account  for  this  finding.  For  the  experiment  reported 
herein,  the  initial  information  (although  it  varied  in  strength)  unam¬ 
biguously  pointed  toward  one  hypothesis  or  another;  only  one  piece  of 
subsequent  information  was  presented  at  a  time;  and  the  subsequent  informa¬ 
tion  was  unambiguous.  None  of  these  conditions  existed  in  tho  Tolcott  et 
al .  studies . 

We  also  found  e  number  of  interaction  effects  involving  either 
response  mode  or  order.  For  example,  we  found  that  the  size  of  the  order 
effect  was  effected  by  both  the  strength  of  the  negative  information  and 
whether  the  aircraft  was  friendly  or  hostile.  Moreover,  this  conclusion  x 
order  x  strength  of  negative  information  interaction  occurred  primarily 
when  Information  waa  presented  sequentially,  not  when  it  was  presented  all 
at  once.  Although  these  third-  and  fourth -order  interactions  accounted  for 
a  very  small  proportion  of  the  variance  in  the  participants '  final  proba¬ 
bility  estimates ,  they  do  indicate  that  order  effects  are  affected  by  more 
than  Just  the  response  mode. 

Poet-hoc  analyses  indicated  that  the  order  effect  was  subject  to 
individual  differences.  In  particular,  anils  ted  personnel  showed  *  larger 
order  affect  than  officers  when  Information  was  presented  sequentially; 
neither  group  shoved  an  order  affect  when  information  was  presented  all  at 
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once.  The  amount  of  eir  defense  experience  did  not  effect  the  results,  tfe 
consider  this  finding  consistent  with  that  of  Messier  end  Tubbs  (1990), 
although  they  concluded  that  there  was  an  experience  aein  effect  by  .. 
accepting  an  alpha  level  of  p  -  0.096  as  significant. 

We  think  the  effect  due  to  rank  la  a  function  of  the  duties  typically 
performed  by  Army  air  defense  officers  and  enlisted  personnel.  In  par* 
titular,  officers  actually  make  the  Identification  and  engagement 
decisions.  Although  enlisted  personnel  monitor  the  situation,  "thalr 
primary  task  is  to  implement  the  engagement  decision.  Therefore,  due  to 
the  requirements  of  their  jobs  and  on* the* job  training,  we  would  hypothe¬ 
size  that  officers  are  less  susceptible  to  order  effects  when  Information 
is  presented  sequentially  and  a  probability  estimate  is  obtained  each  time. 
A  second,  and,  in  our  opinion,  less  likely  hypothesis  is  that  officers  are 
more  likely  to  attend  college  end,  therefore,  obtain  more  training  in 
analytical  thinking  than  enlisted  personnel. 

A  second  finding  was  that  some  of  the  participants  exhibited  large 
order  effects  when  information  was  presented  sequentially,  while  others 
showed  no  order  effects  at  all.  At  one  extreme,  thirty-seven  percent;  <N  - 
11)  of  the  enlisted  personnel  and  nine  percent  (N  -  3)  of  the  officers  not 
only  showed  an  order  effect  for  all  eight  of  the  possible  comparisons,  but 
the  differences  in  their  probability  estimates  was  so  large  that  they 
routinely  switched  their  judgment  as  to  whether  the  unknown  aircraft  was 
friendly  or  hostile,  based  on  the  order  with  which  the  seme  Information  wee 
presented.  In  contrast,  almost  one*third  of  the  enlisted  personnel  end 
almost  one-half  of  the  officers  who  participated  in  the  experiment  showed 
essentially  no  order  effects  when  information  was  presented  sequentially. 
The  latter  results  detract  from  the  universality  of  the  Hogarth- Einhom 
model . 


Two  other  findings  also  detracted  from  the  support  for  the  Hogarth- 
Einhorn  model.  First,  in  contrast  to  the  model's  prediction,  negative 
information  did  not,  on  the  average,  have  a  larger  impact  the  larger  the 
anchor.  Although  there  were  minor  differences,  this  finding  held  for  all 
eight  comparisons  defined  by  the  2  (conclusion)  x  2  (strength  of  first 
piece  of  information)  x  2  (strength  of  negative  information)  portion  of  the 
design.  This  finding  is  at  odds  with  the  empirical  results  obtained  by 
Asere  (1990),  Ashton  and  Ashton  (1988),  and  Einhom  and  Hogarth  (1987). 

Ve  offer  two  hypotheses  to  explain  this  finding.  The  first  hy¬ 
pothesis  Is  that  the  difference  between  the  two  anchors  prior  to  the 
receipt  of  negative  information  was  too  small  in  our  study  to  generate  the 
hypothesized  effect.  Examination  of  Figure  3  shows  that,  prior  to  the 
receipt  of  negative  information,  there  was  e  difference  of  only  0.12  in  the 
mean  probability  estimates  after  the  second  piece  of  information  for  the  FN 
order  (0.62)  and  the  first  piece  of  information  for  the  KP  order  (0.70). 
This  seems  small  to  us.  Moreovsr,  the  largest  differsnee  for  any  of  the 
eight  comparisons  was  only  0.14. 

The  second  hypothesis,  which  could  co-exist  with  the  first  one,  is 
that  participants  were  able  to  use  the  positive  information  in  the  PM  order 
to  generate  an  explanation  for  the  negative  information  and,  thereby, 
lessen  its  impact.  For  example,  if  participants  were  told  (1)  that  the 
aircraft  had  an  IFF  Friendly  Response  and  <2)  was  idsntif led  as  friendly  by 
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higher  headquarters — two  strong  but  not  perfect  predictors  of  friend — then 
they  night  somewhat  discount  the  third  piece  of  information,  which  is  that 
the  aircraft  is  Jamming  friendly  radar,  by  assuming  that  the  aircraft  was  a 
*  disoriented  friendly  jammer” — a  plausible  event.  The  amount  o£  die* 
counting  may  not  have  depended  as  much  on  the  size  of  the  prior  probabil¬ 
ities  or  number  of  positive  pieces  of  lnforaation  in  our  study,  as  on  the 
generation  and  plausibility  of  the  explanation.  The  larger  final  probabil¬ 
ity  estimate  for  the  KP  than  ?N  order  might  occur  because  the  positive 
information  not  only  substantiates  but  exaggerates  the  explanation, 
conaistent  with  the  use  of  an  anchoring  and  adjuatmont  heuristic.  Obvious¬ 
ly,  future  research  is  required  to  test  these  hypotheses. 

The  second  finding  detracting  from  the  support  for  the  Hogarth- 
Einhora  model  ia  that,  cumulatively,  the  order  effects  accounted  for  a 
small  proportion  of  the  total  variation  in  the  participants'  probability 
estimates  (R2  <  0.10).  Moreover,  other  factors,  most  notably  the  "strength 
of  negative  information"  and  the  "conclusion,"  accounted  for  more  variance 
than  did  any  single  effect  due  to  information  order  or  response  .mode . 

Ashton  and  Ashton  (1988;  in  press)  and  Hessiar  (1990)  also  found  the 
strength  of  negative  information  to  be  a  strong  predictor  of  participants' 
probability  estimates.  And  finally,  most  of  the  variance  was  due  to 
subjects  and  not  the  independent  variables  manipulated  In  this  experiment. 
Together,  these  data  suggest  that,  although  they  resulted  In  stetlstlcelly 
significant  findings  consistent  with  the  Hogarth- Einhora  model's  predic¬ 
tions,  information  order  and  response  mode  may  not,  in  total,  be  the  best 
predictors  of  air  defense  operators'  mean  probability  estimates.  Or,  to 
put  it  differently,  the  effects  due  to  information  order  and  response  mode 
may  be  statistically  significant  but  small. 

This  conclusion  must  be  considered  with  ceution  for  three  reasons. 
First,  most  of  previous  research  Investigating  the  impact  of  Information 
order  and  response  mode  has  not  presented  R2  information.  Ashton  (1988;  in 
press),  Einhorn  and  Hogarth  (1987),  Hogarth  and  Einhorn  (1989),  and  Messier 
(1990),  for  example,  do  not  present  ell  the  data  necessary  to  reconstruct 
the  Rzs.  Asere  (1990)  does,  however,  present  enough  date  in  the  ANQVA 
tables  for  his  two  experiments  to  calculate  R2*.  The  R2  for  the  sig¬ 
nificant  order  effect  in  his  first  experiment  was  0.22;  it  was  0.067  in  the 
second  experiment. 

Second,  the  error  terms  in  the  current  experiment  were  no  doubt 
inflated  by  (1)  the  artificialities  of  our  paper- end-pencil  task,  and  (2) 
the  fact  that  air  defense  operators  do  not  make  probability  estimates  as 
part  of  their  job.  The  Inflated  error  terms  certainly  reduced  the  amount 
of  systematic  variation  in  the  participants'  responses  that  could  be 
accounted  for  by  information  presentation  and  response  mode.  Indeed,  under 
these  circumstances,  one  might  argue  that  accounting  for  tan  percent  of  the 
variance  Is  quite  good. 

Third,  aa  Christensen- Szalanakl  and  Fobian  (in  press)  point  out  In 
their  nets  -analysis  ©f  the  hindsight -bias  literature  ,  determining  whether 
an  effect  site  la  of  practical  significance  or  utility  is  situation- 
specific.  Discussion  with  senior- level  Army  eir  defense  personnel 
indicates  that  the  results  may  have  implications  for  (1)  training  air 
defense  operators,  and  (2)  designing  information  displays  for  future  eir 
defense  systems.  The  esse  would  bo  considerable  stronger,  however,  if  the 
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results  vers  obtained  (1)  using  actual  air  defense  s isolators,  and  (ii)  if 
we  had  investigated  Whether  changes  in  the  probability  estimates  also 
affected  behavior,  vhich,  in  this  case,  vould  be  the  decision  to  engage 
aircraft  identified  as  hostile.  Each  point  ia  considered  briefly,  in  turn* 

First,  the  genexalizability  of  the  obtained  results  is  the  critical 
issue  driving  the  desire  for  simulators.  The  issue  of  general  liability  is 
not  unique  to  our  study,  for  all  the  studies  referenced  herein  have  used 
paper-and*pencil  tasks.  Whether  such  representations  adequately  addressed 
the  issue  of  generalizability  depends  on  their  task  domain.  Second,  there 
is  minimal  research  investigating  the'  relation  between  order  effects  and 
behavior.  Asare  (1990)  has  shown  that  Information  order  can  affect 
auditors'  likelihood  estimates  and  behavior.  However,  Meisser  (1990)  found 
the  effect  on  probabilities — but  not  behavior — using  a  different  auditing 
task. 

Finally,  the  current  study  suggests  thst  care  be  used  in  employing 
knowledge 'engineering  procedures  to  develop  expert  systems.  Knowledge 
engineering  with  trained  personnel  can  proceed  in  an  interview  format  that 
emphasizes  the  sequential  presentation  of  information  and  usea  paper -and - 
pencil  tasks  analogous  to  those  used  in  the  current  experiment  (e.g.,  see 
Forsythe  and  Buchanan,  1989).  As  Lehner  and  Adelman  (1987;  in  press), 
Hammond  (1987),  and  von  Winterfeldt  (1988)  note,  most  knowledge  engineers 
are  not  familiar  with  the  cognitive  heuristics  and  biases  literature  and 
may,  inadvertently,  encode  biases  into  the  knowledge  bases  of  expert 
systems.  The  results  of  the  current  effort  Indicate  that  knowledge 
engineers  should  use  multiple 'information  orders  and  response  modes  to 
assess  the  consistency  of  obtained  probability  estimates.  Ideally,  they 
would  then  use  case  data  with  the  correct  answers  to  assess  the  predictive 
accuracy  of  the  elicited  knowledge. 
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